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What is S&MA? 


S&MA is... Program/Project Management! 

S&MA is... Systems Engineering! 

S&MA is... Real Engineering! 

S&MA is... a framework of methodologies, analyzes, tools, and processes: 

• For the meaningful organization of complex data and information 

• For the successful execution of critical tasks 

• And ultimately for the proactive management of risks and margins to 
achieve desired results. 


S&MA, Six Primary disciplines: 



Works when you want it to, 
Doesn’t work when you don’t want it to, 
Provides capability in a contingency 


S&MA: 


• System Safety 

• Reliability and Maintainability 


K Quality Engineering 
• Software Assurance 




j . 


• Operational Safety 
^ Aviation Safety 
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What is System Safety? 


System Safety engineering specificaiiy addresses the identification, anaiysis, 
and controi of system risks to humans, the environment, and mission assets. 

The System Safety assessment inciudes numerous forms of high-ievei anaiyses 
performed to support safety decisions such as: 

• Hazard Anaiysis 

• Probabiiistic Risk Assessment 

• System Safety Performance Anaiysis 

• Phenomenoiogicai Anaiysis 
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What is System Safety? 


SMA lithdiCHl EMCEntliet Pro i ram 


Basics of System Safety 



▼ COURSE MAP 


Key Concepts and Analysis Techniques > Risk Reduction Protocol 



Risk Reduction Protocol 


Follow MPR S715.3 to eliminate the danger, reduce the probability of it happening, or reduce the impact. 

Project managers shall ensure that hazards and dominant contributors to risk are controlled according to the following: 

a) Eliminate accident scenarios (e.g., eiiminate hazards or mitioting events by design]. 
bj Reduce the likelihood of accident scenarios through design and operational changes [hazard control], 

c ) Red u ce t h e severity ofaccident consequences [hazard mitiga tion]. 

d) Improve the state-of- knowledge regarding key uncertainties that drive the risk associated with a hazard 
(iiflcertcf/nfy reduction to support implementation of the above strategies]. 


Glossary Help 


PAUSEII «BACK NEXT» 2 of 9 REPLAY d CC 
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What is Reliability and 
Maintainability? 


Through design evaluation, probabilistic modeling and analysis, and testing, 
the Reliability and Maintainability disciplines help establish the necessary 
confidence that the system and its components will function as required. 


This discipline is split into two parts: 

1. Reliability Engineering: assessment and improvement of reliability 
performance of systems during their missions. 


Reliability: The system performs as intended when needed 


2. Maintainability Engineering: assessments and verification of the system 
design characteristics so that downtime and the need for maintenance are 
minimized. 


Maintainabiiity: How fast, easy, and safe it is to 
repair the system when necessary 
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What is Reliability and 
Maintainability? 

f ^ ^ 



AVAILABILITY 

When you want to use 
your car, is it available 
for your use? 


RELIABILITY 

How often does your 
car fail to start? 
How often do you have 
trouble with your car? 


J 


MAINTAINABILITY 

How often does your 
car need repair? Can 
maintenance be done 
quickly and easily? 

L J 



R&M performance 
metrics are based on 
future events; Never 
100% accurate 



R&M relies on past 
experiences, for example 
past mission 

performance or test data 


r 



R&M assesses ways in 
which the system fails, 
and methods by which it 
is restored 
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What is Reliability and 
Maintainability? 



Design Engineer: Success Space 

How will the system work? 
What function will it perform? 
What are the requirements? 




Reliability Engineer; Failure Space/ 
Worst Case Failure Mode 

How can the system fail? 

How can the operating environment 
cause problems? 

Is redundancy required? 

Are there operational work-arounds? 
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What is Reliability and 
Maintainability? 


Examples of Qualitative 
Program Refiablfity 
Requirements 

Failure Tolerance 


Single Point Failures 


Reliability Analyses 


Examples of Quantitative 
Program Reliability 
Requirements 

Failure-Free 
Performance Goal 


Probability of 
Success 
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What is Reliability and 
Maintainability? 


R&M Tools and Methodology 


FMEA is a bottom-up analysis of failure modes whereas Fault tree logically decomposes system failures into contributing anomalous 
events and conditions. Example taken from James Webb Space Telescope (JWST). 


FMEA Example 
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What is Quality Engineering? 





Quality [quaH'ty;) degree to which a set of inherent 
characteristics fulfills requirements." 

Quality Engineering includes the design, compliance, and fitness for use of its products and 
services. As part of the overall Quality Assurance effort, it serves to provide confidence that 
product configurations meet safety and technical requirements. Quality Engineering forms an 
essential part of the overall plan to achieving safe and successful missions consistently and 
continuously. 

Achieving Quality Engineering requires: 

• Establishing needs and expectations 

• Developing an effective quality management process 

• Establishing engineering and manufacturing practices that emphasize robust design, the 
state where the technology, product, or process performance is minimally sensitive to 
factors causing variability 

Identifying critical processes , processes that, if performed incorrectly or in violation of 
prescribed requirements, could result in loss of life, serious personal injury, loss of mission, 
or loss of a significant mission resource 

Identifying key characteristics , the features of a material, process, or part whose variation 
has a significant influence on product fit, performance, service life, or manufacturability 
Verifying that the product, as built, meets the design 

Developing process maturity through continuous process improvement efforts 


Example: Tile Repair, “Bubbles in the Goo” 
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What is Quality Engineering? 


■ MTntoilvilEmil 



Basics of CtLialltv Engineering 


QE Overview > Why cantinuously «trive for better (quality? 

If the shuttle h£]$ approximately 24^300 tiles adhered to its 
exterior surfaces. A tile installation process at: 

^ 9938% f Four Sigma Level): approximately 150 tiles not being 
glued on properly. 

99.99966% (Six Sigma Level): approximately 0.08 tiles not being 

glued on properly 



Good 


bnough s 
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What is Software Assurance? 


The Software Assurance discipline operates under the scrutiny of a planned and systematic set 
of activities that ensure that the software and its related products: 

• Conform to software life cycle processes 

• Meet their specified requirements and standards 

• Are consistent, complete, correct, and safe 

• Are secure and reliable as warranted for the system and operating environment 

• Satisfy customer needs 

• Are implemented according to plan 


In order to achieve these goals, the Software Assurance discipline consists of five distinct roles: 

• Software Quality: assurance that quality is built into the software 

• Software Safety: an approach to identifying, analyzing, and controlling software hazards 

• Software Reliability: an approach to incorporating and measuring reliability throughout the 
product lifecycle by building in software error prevention, fault detection, isolation, recovery, 
and/or reduced functionality states 

• Software Verification & Validation (V&V): activities which ensure that software satisfies 
functional requirements and that each phase of the development process yields acceptable 
products 

• Independent Verification & Validation (V&V): additional V&V activities performed by an 
independent organization 


Software has become an increasingly more significant, more 
complex, and more critical part of integrated space systems... 
and therefore software assurance has also grown significantly! 
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What is Operational Safety? 


Managing risks and preventing operational accidents is imperative for SMA’s 
Operational Safety workforce. This NASA SMA function focuses on the 
prevention of operations-related safety hazards by: 

• Assuring mission success 

• Protecting the public and flight, ground, laboratory, and underwater personnel 

• Protecting the environment 

• Protecting the aircraft, spacecraft, and payloads 

• Protecting the facilities, property, and equipment 

The governing policy directive for this discipline is the NASA General Safety 
Program Requirements (NPD 8715.3) Chapters which specifically focuses on 
the following key aspects: Motor Vehicle Safety; Personal Protective 
Equipment; Control of Hazardous Energy (Lockout/Tagout Program); Pressure 
System Safety; Electrical Safety; Hazardous Material Transportation, Storage, 
and Use; Hazardous Operations; Laboratory Hazards; Lifting Safety; Explosive, 
Propellant, and Pyrotechnic Safety; Underwater Operations Safety; Launch, 
Entry, and Experimental Aeronautical Vehicle Operations Safety; Test 
Operations Safety; Non-Ionizing Radiation; Ionizing Radiation; and. Confined 
Spaces. 

Additionally, there are many Federal, State, and Local laws that also apply to 
Operational Safety at NASA. 
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What is Aviation Safety? 

...assure mission success and preserve human and materiai resources 


Structural and material changes that alter the basic aircraft design 
Modifications of the exterior contour or mold lire of the aircraft 
Modification to the flight control system, including software revisions 
New or modified propulsion system or its control system 

Modifications of any subsystem interfacing with and affecting flight or propulsion systems 

Modification of the aircrew life support systems 

Flight test instrumentation that interfaces with normal aircraft systems 

Intentional operation in a degraded mode for test purposes 

Dropping of uncertified stores or objects 

Any other modifications, payloads, or operations that are nonstandard according to established flight manuals 


Examples of Changes Requiring an Airworthiness Review 


Readiness Reviews 

Airworthiness, flight safety, and mission readiness reviews are conducted for all aircraft modifications. 

These reviews are to clear unique or nonstandard internal or external payloads or stores 
configurations for flight and to review nonstandard flight operations. 

These reviews identify hazards so as to minimize risks to persons and property and to enhance the 
likelihood of mission and program success. 


Formal review requirements are tailored according to the type of modification incorporated specific 
to the mission and the operational risks involved. 


• Indentify, Analyze, Eliminate, and 
Report Hazards 

• Risk Assessment / Risk Management 

• Fault Tree Analysis 

• FMEA/CIL & FMECA 
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What is 

An Integrated Discipline! 



Hazard Analysis^ 


Failure Modes 
Effects Analysis 


PRA 


Reliability 

Anaiysis 


T 


Problem 

Reporting 




Risk 

Management 

4 


Trend Analysis 


Six primary S&MA disciplines: 

^ System Safety 

Reliability and Maintainability 
Quality Engineering 
Software Assurance 
Operational Safety 
Aviation Safety 
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Percent of Our Effort 


What is 

The “Psychology” Side of S&MA 


Where we 
excel as 
‘engineers 
and 

problem 

solvers 


Where we 
struggle as 
“human 
beings”, 
tough to 
sustain!! 



Year 1 Year 2 Year 3 Year 4 Year 5 Year 6 

Time Since Major Incident 


Year 7 
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What is 

The “Softer/People” Side of S&MA 



Bryan O’Connor 
Agency Chief, S&MA 


Bryan O’Connor’s Characteristics of a Great 
S&MA Professionai: 

• Technically Credible 

• Imbued with “Engineering Curiosity” 

• Courageous (“truth to power”) 

• High Integrity 

• Soiid Knowiedge of Requirements and Rationaie 

• Good Communication Skiiis (Verbai & Written) 

• Experience in Appiicable Fieid 

• Humble Yet Engaged 

• Persistent Yet Pragmatic 

• Energetic and Creative (Yes, if....) 

• Thick Skin and Sense of Humor (for Longevity) 


“We're just flat not as smart as we think we are” 

Tommy Holloway 

Space Shuttle Program Manager 
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Birth and Sustainment 

Of US “Assurance”? 



“No man is allowed to be a judge in his own cause, because his interest 
would certainly bias his Judgment, and, not improbably, corrupt his integrity. 
With equal, nay with greater reason, a body of men are unfit to be both Judges 

and parties at the same time;” 

The Federalist No. 10 

(a series arguing for the ratification of the United States Constitution) 

James Madison 
November 23, 1787 


“Trust, but verify” 

Adopted and made famous by U.S. president Ronald Reagan. Reagan frequently used it 
when discussing U.S. relations with the Soviet Union. Reagan rightly presented it as a 
translation of the Russian proverb "doveryai, no proveiyai" (Russian: AoBepnM, ho 
npoBepHM). Soviet revolutionary Viadmir Lenin also frequently used the phrase. Reagan 
used the phrase at the signing of the INF Treaty, his counterpart Mikhail Gorbachev 
responded: "You repeat that at every meeting," to which Reagan answered "I like it.“ 


George K. Gafka 281-483-7732 


27 




Legend 


Program/Project S&MA Interfaces 
Organizations and People, “Governance” 
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S&MA Throughout the Program Lifecycle 
Modeled after NPR7120.5 

NPR 7120.5, NASA Space Flight Program and Project Management Requirements 


NASA Life Cycle Highlighting System Safety Products and Services 


A general overview of 
the activities, processes, 
and products associated 
with System Safety 
engineering within the 
NASA Project Life Cycle 


\ 
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Program/Project S&MA Interfaces 
S&MA Content Example: NPR8705.2b 

NPR 8705.2B Human-Rating Requirements for Space Systems 



Figure 1 - Agency Requirements Framework Related to Human-Rating 
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Program/Project S&MA Interfaces 
S&MA Content Example: NPR8705.2b 

NPR 8705.2B Human-Rating Requirements for Space Systems 



Figure 2 - Relationship Among Requirements 
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Program/Project S&MA Interfaces 
S&MA Content Example: NPR8705.2b 

NPR 8705.2B Human-Rating Requirements for Space Systems 
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Figure 3 - Human-Rating Certification Process Flow 
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Interesting Relative Risks 

Example: Agency Loss Of Crew (LOC) 


U.S. combatants in the battle of Iwo Jima 1945 
Doolittle raid1942, and Pickett’s charge 1864 
U.S. combatants on D-Day 
New York City firefighter on 9/11 
B-17 single mission over Germany 1943 
Mt Everest climb (1922 - 2006) 

Soyuz missions (manned flight statistics 1967 - present) 
Space Shuttle mission (statistics 1981 -present) 

Space shuttle mission to/fm ISS (2010 PRA) 

Cx and CCT aaencv threshold (sinqle ISS mission) 

X-15 research flight 

Cx and CCT desiqn requirement (sinqle ISS mission) 
Alaskan crab fisherman (one year) 

U.S. crop duster pilot (one year) 

Cx and CCT recommended qoal (sinqle ISS mission) 

U.S. logging, timber cutting (one year) 

U.S. construction worker (one year) 

U.S. coal miner (one year) 

1/10* 

1/11* 

1/29* 

1/34* 

1/37** 

1/49* 

1/52** 

1/66** 

1/89** 

1/150** 

1/199** 

1/270** 

1/281* 

1/510** 

1/750** 

1/775* 

1/2440* 

1/3450* 

* deaths/total participants 
** fatal mishaps/total missions 
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Example: Shuttle Program, fixes & improvements 


SPACE SHUTTLE PROGRAM 

Space Shuttle Safety and Mission Assurance Office 
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S&MA as a Function of Organization 

Program/Project S&MA Relationship (“tightly coupled”) 


Program / Project 
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Value needed and proactive value added 
Healthy tension and checks & balances 



Program / Project 




V 


JSC S&MA 




S&MA 


/ 0 0 U 0 0 



Independent, but informed? 

Conscious of Program/Project, relevant? 
Value needed? Value added? 

Checks & balances, but the right areas? 


0 


3 Jobs of S&MA: Doing... Checking... Technicai Authority 
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S&MA as a Function of Engagement Timing 
Engagement Example: NPR7123.1 

NPR 7123.1, NASA Systems Engineering Processes and Requirements 


S&MA Breadth of Initial/Significant Engagement 


[ 


- Tailoring reqs and processes 

- Informing risk trades 

- Proper scoping 


NASA Lllfr 
Cycle Ph»e» 

Pi-darSfll LIlie 
Cycle Oetce S 
Major Events 


Profecl 
Slait3 

Program 
UpdatBE 

Agency Review^ 


Major Program 
Revifiwa' 



FORMULATION 


Iri rfan 




FAD 


Zi 


A 

ASP 


KDP 


fy\/' 



KPP 

PCA1^ 

Program j 




- Compliance to what's on the books 
-Auditing “water under the bridge” 

- Reactionary resourcing 


IMPLEMENTATION 

PfogfStn Actfuiiilicwt O^erati’oi'js 




ASM 


I 

A ^ 

(PPAR'> 

UncDupledlfii Loosely Coupled Programs 
or 

Single-Profect^ S Tightly Coupled Programs' 
Figure 5.1 The NASA Program Life Cycle 



ASlart Proj*<^ 3 V,^ „ „ . - 
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S&MA as a Function of Resources 



^ S&MA Breadth of Capabilities/Services ^ 


Contributing to Leading 

Compliance with Program/Project Program/Project 

Compiiance Program/Project Risk Reduction and Risk Reduction and 

with laws Safety Reqs Mission Success Mission Success 



George K. Gafka 281-483-7732 


38 



8 o, how much shou l d S &MA “cost”? 



How much should be “invested” in S&MA? 


Core tenets: 

• S&MA is an “investment”, not a 
“cost.” S&MA enabies safety 
and mission success... and 
actually saves money! 

• S&MA is an Agency, Program, 
Project, or Center risk mitigation 
strategy against safety, mission 
success, scheduie and cost (cost 
of quality) threats 


Program/Project S&MA scope? 

• Large? Small? 

• Complex? Simple? 

• Critical functions? 

• Critical hazards? 

• Make vs. buy? “In-line”? 

• InsIght/oversIght model? 

• “Human” spaceflight? 

• Who is accountable? 

• Multiple NASA centers? 

• Multiple contractors? 

• International Partners? 

• TRL level? objectives? 

• Acquisition phase/maturity? 

(SRR? PDR? CDR? Ops?) 


Notionaiiy represents one comparative snapshot in time, changes over lifecycle! 
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Demonstrated Value of “Assurance” 

Historical Example 


Early Mission Reliability (First 3) 


1 — I — I — I — I — I — I — I — I — I — I — I — I — I — I — I — rn — I — I — rri — r 
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1 

Comm-Eretal ELV 
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Calendar Year 

* Design certification effect on early mission reliability is significant 

- 97.0% vs 69.9% re//ajb///fy - heritage vs. commercial 

- 96.2% vs 69.9% re//ajb///fy - EELV vs. commercial 

* Order of magnitude difference in failure rates 

- 3% vs 30% failure rates 

100% of faifuros were commercial launches 
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Proven Reliability 

Performance - Flight History 


Historical Account of “Human Rating” 

We’ve Gotten There Multiple Ways 


Mm @f M(mpt&UsSi 

(at time of acceptance for 1" use by U.S. Astronauts) 




Launch Vehicle # 
Spacecraft A 
Uncertainty 


Process Confidence 

Insight - Influence - Control - Oversight - Test & Verification 
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S&MA’s Challenges 


High Reliability Organizations 
Challenges at NASA 

• Advanced Technology 

• Advanced, leading edge technology, difficult to intellectually manage 

• Allowable Failure Rate 

• Fewest number of failures allowed to be considered successful 

• High Visibility 

• Intense media coverage, public interest 

• Organizational Complexity / Size / Diversity 

• Highest number of decisions and people involved per event 

• Research and Development 

• Don’t always have answers; Independent Safety 

• excerpt from presentation on “High Reliabiiity Organizations” 



Seek 


Guidance' 
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Modern-Day Guidance For “Transportation” 

Do We Set Ourselves Up For “Failure”? 






Mission Directorate: Expioration Systems 

Theme: Commerciai Spaceflight 

Program: Commercial Crew Program Overview 


“Failure” meaning: 
performance # expectations 


[919 


The Commercial Crew Program will provide $6 billion over the next five years to support the development of commercial crew 
transportation providers to whom NASA could competitively award a crew transportation services contract analogous to the Cargo 
Resupply Services contract for ISS. 

These funds will be competed through COTS-like, fixed-price, milestone-based Space Act Agreements that support the 
development, testing, and demonstration of multiple commercial crew systems. As with the COTS cargo program, some amount of 
private investment capital will be included as part of any Space Act Agreement and NASA will use this funding to support a range of 
higher- and lower-programmatic risk systems . Unlike the COTS program, which exclusively funded entirely new and integrated 
systems (launch vehicles plus capsules), this program will also be open to a broad range of commercial proposals including, but not 
limited to: human-rating existing launch vehicles, developing spacecraft for delivering crew to the ISS that can be launched on 
multiple launch vehicles, or developing new high-reliability rocket systems. 

NASA will leverage existing COTS and Commercial Crew Development (CCDev) activities to engage a broad spectrum of private 
industry, from emerging to established companies, with a full and open competition for commercial development activities at the 
conclusion of the CCDev activities. The competition will result in a targeted portfolio of up to four companies with a mixed risk 
balance consisting of launch vehicles, crew capsules, and supporting technologies , similar to the Commercial Crew Development 
awards from Recovery Act funds announced on February 2, 2010. The number of awardees will be based on such factors as 
technical competency and available funds. Firm-fixed-price awards will be issued for production of crew services after a key 
progress review of the down-selected commercial companies as necessary, within the available budget. 

At no point in the development and acguisition of commercial crew transportation services will NASA compromise crew safety . 
NASA has unique expertise and history in this area, and a clearly demonstrated record of success. NASA will bring that experience 
to bear in the appropriate wav to make sure that commercial crew transportation services are a success both programmatically, and 
with respect to safety. In that regard, NASA agrees with the Aerospace Safety Advisory Panel, which stated, “it is crucial that NASA 
focus on establishing the certification requirements, a certification process for orbital transportation vehicles, and a process for 
validating compliance. The performance and safety requirements must be stated promptly and clearly to enable NASA and non- 
NASA entities to proceed in the most productive and effective manner possible.” NASA will work to complete an agency and 
industry-coordinated human rating draft by the end of 2010. 


http://www.nasa.gov/pdf/428356main_Expioration.pdf 
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Historic Guidance For “High Risk Exploration” 

Today: Are We Too Risk Averse To “Explore” More Cheaply? 

Are We Too Cheap To Buy The Risk Posture We Say We Want? 

Or, is there some new (elusive) way to achieve better S&MA for less $? 


“To your own discretion therefore must be left the degree of danger you 
risk, and the point at which you should decline, only saying we wish you 
to err on the side of your safety, and to bring back your party safe even if 

it be with less information. ” 

Thomas Jefferson Letter to Meriwether Lewis: 1803 


During the "Heroic Age of Exploration," the 
period in which Shackleton's 1914-1916 
British Imperial Trans-Antarctic Expedition 
took place, Antarctic expeditions often 
became ordeals of suffering. At the time, 
polar explorers were revered for their 
sacrifices and held up as heroes, albeit 
often tragic ones. Shackleton handpicked 
some members, to recruit the rest, it is said 
that he posted the following notice: 


lifter Cold 

vpl darJt of oc ui&Ie t n 

' 
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Thank you, 
Onward and Upward! 
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